1 First Steps

1.1 What is Google Colab?

Google Colab works by allowing users to write and execute Python code in a Jupyter notebook environment hosted on the cloud. It provides free access to computing resources, including CPUs, GPUs, and TPUs, which can significantly accelerate computational tasks. Users can create, share, and collaborate on notebooks in real-time, making it an excellent tool for educational purposes, data analysis, machine learning, and research. Colab also integrates seamlessly with Google Drive, enabling easy saving and sharing of projects.

1.2 How do I access Google Colab?

The first thing is to have a Google account, and to access Google Colab, it’s as simple as opening a web browser, going to https://colab.google/, and clicking the “Open Colab” button. This will open a new page where you can create a new Google Colab document.

1.3 What is Tensorflow?

TensorFlow is an open-source machine learning framework developed by Google. It allows users to build and train various machine learning models, ranging from simple to complex, using computational graphs. TensorFlow is widely used for tasks such as classification, regression, neural networks, and more, across different platforms and devices.

2 Training a TensorFlow model in Google Colab

Next, we will explain how to train a TensorFlow model in Google Colab to detect objects, specifically distinguishing between wild-type flies and white-type flies. Also, in the following sections, datasets will be provided for labeling or a dataset already labeled and ready to be uploaded to Drive and used for training.

2.1 Dataset preparation

First of all, you need a set of photographs that include the flies. To capture the pictures, we used a Bysameyee 8-SA-00 digital microscope, which provides more detailed photos. Additionally, we used a Canon EOS 70D camera with a 100mm macro objective f/2.8L lens in the experiment. We also used cardboard of different colors as backgrounds, which helped to improve accuracy. The images from the Canon camera were quite large and couldn’t be processed by the model, so they were divided into equally sized fragments using a Python code and the photographs were taken on a printed grid:

import image_slicer
from PIL import Image
path = "" # add the full path of the image
n = 20 # number of pieces
image_slicer.slice(path, n)

The captured images in this format, which can be used for the tutorial, are available at the following link:

drosophila_dataset

After this, the images will be divided into three different folders using another Python code:

import splitfolders
folder = "" # Add the address of your folders
splitfolders.ratio(folder, output="output", seed=1999, ratio=(.8, 0.1, 0.1))

This code will divide the images into 3 folders: 80% of the total images will be used for training the model, 10% for validation during training, and 10% will be reserved for external checks.

2.2 Labeling

The last step in dataset preparation was to define what the AI needed to learn. This was done using the open-source program LabelImage (https://github.com/tzutalin/labelImg). The program allows flies to be labeled and classified as wild type or white type in a graphical interface. An .xml file is generated in pascal voc format, which records the label and coordinates of the flies in the image. This labeling process is done for all images in the training and validation folders.

2.3 Increased training efficiency (optional but strongly recommended):

To enhance the training performance, two techniques were applied to the dataset using the roboflow web platform (https://roboflow.com). The first technique involved automatically combining image fragments to improve the detection of small targets. The second technique involved artificially introducing noise into the photos to enhance their resilience to photo artifacts. An example of this is the following image with 5% noise.

To follow the tutorial, you can use the following labeled dataset:

labeled_drosophila_dataset

2.4 Google Colab preparation for training models

In the following link, there is already a Google Colab prepared to train Tensorflow models, but here we will explain it step by step.

The first step is to change the runtime type because we will need to use a GPU. To do this, click on Runtime > Change runtime type at the top, and a menu will open where you can select a GPU.

2.4.1 Mount google drive on Google Colab notebook

In a code block, you should add the following lines:

from google.colab import drive
drive.mount("/content/gdrive")

When you run this code block with the play button, it will ask for some permissions, and after granting them, you will have a connection between the Google Colab notebook and Drive. Finally, we connect to Google’s resources by clicking the connect button.

2.4.2 Virtual environment preparation

When connecting to the environment, Google Colab already has numerous libraries loaded for both Python and R. However, in our case, to avoid conflicts between packages, we are going to create a virtual environment using Miniconda. To do this, we need to add the following lines to a code block and execute it:

%env PYTHONPATH = # /env/python
!wget https://repo.anaconda.com/miniconda/Miniconda3-py38_4.12.0-Linux-x86_64.sh
!chmod +x Miniconda3-py38_4.12.0-Linux-x86_64.sh
!./Miniconda3-py38_4.12.0-Linux-x86_64.sh -b -f -p /usr/local
!conda update conda -y
import sys
sys.path.append('/usr/local/lib/python3.8/site-packages')
!conda create -n tensorflowenv python=3.8 -y

This will create a virtual environment called tensorflowenv with Python 3.8. Next, in another code block, we will insert the necessary packages and install them by running the code cell:

%%shell
eval "$(conda shell.bash hook)"
conda activate tensorflowenv
pip install tflite-model-maker==0.3.4 tflite-support==0.3.1 tensorflow==2.6.0 keras==2.6.0 tensorflow-estimator==2.6.0 ipykernel pycocotools
conda install -c conda-forge cudatoolkit=11.2 cudnn=8.1.0 -y

Although the package versions are old, they are the ones we have tested and confirmed to work.

2.4.3 Generating the Python script and execute it

To generate the Python script, in a cell we will include the following lines of code for Import of necessary libraries:

%%writefile  training.py
import numpy as np
import os



from tflite_model_maker.config import ExportFormat, QuantizationConfig
from tflite_model_maker import model_spec
from tflite_model_maker import object_detector

from tflite_support import metadata

import tensorflow as tf
assert tf.__version__.startswith('2')

tf.get_logger().setLevel('ERROR')
from absl import logging
logging.set_verbosity(logging.ERROR)

Using %%writefile training.py creates a temporary script named traing.py.

Continuing with the script, we add the remaining lines of code that will perform the training, pre-exportation validation, and post-exportation validation of the model:

%%writefile  -a training.py
# pathways and name of tensorflow lite model
training_path = "/content/gdrive/MyDrive/drosophila_white_wild_v7_V2/training"
validation_path ="/content/gdrive/MyDrive/drosophila_white_wild_v7_V2/validation"
tflite_filename="drosophila_lite2_epochs120_batch16_img1251_wild_white_v7_V2.tflite"

# Selecting the model architecture
spec = model_spec.get('efficientdet_lite2')

# data upload
training_data = object_detector.DataLoader.from_pascal_voc(
    training_path,
    training_path,
    ["white type", "wild type"]
)
validation_data = object_detector.DataLoader.from_pascal_voc(
    validation_path,
    validation_path,
    ["white type", "wild type"]
)


# Training
model = object_detector.create(training_data, model_spec=spec, epochs=120, batch_size=16, train_whole_model=True, validation_data=validation_data)

# Export model
model.export(export_dir=".", tflite_filename=tflite_filename)


# Model evaluation
print( model.evaluate(validation_data))

# Exported model evaluation
print(model.evaluate_tflite(tflite_filename, validation_data))

Using %%writefile -a training.py adds the remaining lines of code to the temporary script.

In Google Drive, you should have the folders specified in this part of the code with the dataset:

training_path = "/content/gdrive/MyDrive/drosophila_white_wild_v7_V2/training"
validation_path ="/content/gdrive/MyDrive/drosophila_white_wild_v7_V2/validation"

The variable tflite_filename refers to the name that the trained model will have.

The part of the code where the architecture to be trained is specified is:

# Selecting the model architecture
spec = model_spec.get('efficientdet_lite2')

There are 5 different types of models, some smaller and faster, and others larger and slower for performing inference. Additionally, the larger ones require more GPU resources:

Types of models
Model	Size.MB.	Latency.ms.	Average.Precision
EfficientDet-Lite0	4.4	37	25.69%
EfficientDet-Lite1	5.8	49	30.55%
EfficientDet-Lite2	7.2	69	33.97%
EfficientDet-Lite3	11.4	116	37.70%
EfficientDet-Lite4	19.9	260	41.96%

In this section, the training and validation data are loaded. Here, the labels must match those specified in the labeling section:

# data upload
training_data = object_detector.DataLoader.from_pascal_voc(
    training_path,
    training_path,
    ["white type", "wild type"]
)
validation_data = object_detector.DataLoader.from_pascal_voc(
    validation_path,
    validation_path,
    ["white type", "wild type"]
)

Here, the labels are white type and wild type.

To configure how the training will be performed, the following lines of code must be modified:

# Training
model = object_detector.create(training_data, model_spec=spec, epochs=120, batch_size=16, train_whole_model=True, validation_data=validation_data)

Usually, only the values of the following variables need to be modified:

Epochs: An epoch refers to one complete cycle through the entire training dataset. During training, the model processes all the training data once per epoch. Multiple epochs are often used to improve the model’s performance, as each epoch allows the model to learn and adjust its parameters based on the data.
Batch Size: Batch size is the number of training examples utilized in one iteration of training. Instead of processing the entire dataset at once, the training data is divided into smaller batches, and the model updates its parameters after each batch. A smaller batch size typically requires more iterations to complete one epoch, while a larger batch size reduces the number of iterations but requires more memory.

In the section of the code detailed below, the model is exported and validated before and after being exported:

# Export model
model.export(export_dir=".", tflite_filename=tflite_filename)


# Model evaluation
print( model.evaluate(validation_data))

# Exported model evaluation
print(model.evaluate_tflite(tflite_filename, validation_data))

The training results are analyzed using COCO metrics, which will give us an idea of the object detection accuracy. For this purpose, it uses the images from the validation folder.

In a new code block, we will use the virtual environment tensorflowenv created earlier to execute the temporary script:

%%shell
eval "$(conda shell.bash hook)"
conda activate tensorflowenv
python3 training.py

2.4.4 Export model

To export the model from Google Colab, you can do it with the following code:

from google.colab import files
tflite_filename="drosophila_lite2_epochs120_batch16_img1251_wild_white_v7_V2.tflite"
files.download(tflite_filename)

The name of the model established previously must match with tflite_filename.

2.5 Trained models

Numerous models have been trained with different datasets as can be seen here. However, the five available architectures trained with our most longest dataset are located here. The metrics for these can be seen here.

3 Image processing with pre-trained tensorflow model

In the following link, there is already a Google Colab prepared to to process images with Tensorflow models, but here we will explain it step by step.

3.1 Drive preparation

For the correct operation it is necessary that you have a folder called ‘Image_Processing_Approach’ in your google drive with the image you want to process and the trained model.

For example, you can use the following image:

And one of the trained models such as drosophila_lite2_epochs120_batch16_img1251_wild_white_v7_V2.tflite

3.2 Google Colab preparation for image processing with a tensorflow model

3.2.1 Mount google drive on Google Colab notebook